Papers with NLI datasets
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent studies have shown that even state-of-the-art pre-trained language models can generate inconsistent summaries in more than 70% of all cases. |
| Approach: | They propose a method that enables NLI models to be used for inconsistency detection by segmenting documents into sentence units and aggregating scores between pairs of sentences. |
| Outcome: | The proposed method achieves state-of-the-art accuracy of 74.4% on six large inconsistency detection datasets. |
Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in transfer learning have brought significant improvements to many natural language processing tasks. |
| Approach: | They propose a method of fitting a GBDT head on the features computed during finetuning to increase performance without additional computation by the neural network. |
| Outcome: | The proposed method improves on several NLI datasets using a strong baseline model (RoBERTa-large) with MNLI pretraining. |
Developmental Negation Processing in Transformer Language Models (2022.acl-short)
Copied to clipboard
| Challenge: | Negation is an important construct in language for reasoning over the truth of propositions, garnering interest from philosophy (Horn, 1989) and psycholinguistics (Zwaan, 2012). |
| Approach: | They propose to frame a natural language inference task as a problem and examine how well transformers can process negation categories. |
| Outcome: | The proposed models perform better on certain categories, suggesting clear differences in how they are processed. |
Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) datasets often contain hypothesis-only biases . authors propose probabilistic methods to build models that are more robust to such artifacts - a new study shows . |
| Approach: | They propose probabilistic methods to build models that are more robust to biases . authors train on datasets containing biase . |
| Outcome: | The proposed methods can make NLI models more robust to dataset-specific artifacts . the methods transfer better than a baseline architecture in 9 out of 12 NLI datasets compared with baseline architectures based on the proposed methods . |
Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to natural language inference focus on interaction architectures of sentences . but, we propose to transfer knowledge from discourse markers to augment the model . |
| Approach: | They propose to transfer knowledge from discourse markers to augment the quality of the NLI model. |
| Outcome: | The proposed method achieves state-of-the-art performance on large-scale datasets. |
Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization (2021.naacl-main)
Copied to clipboard
Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, Kartik Talamadupula
| Challenge: | Large-scale, open Natural Language Inference datasets have catalyzed the development of NLI models that exhibit close to human-level performance, but the use of these models for other downstream NLP tasks has met with limited success. |
| Approach: | They use multiple-choice reading comprehension and checking factual correctness of textual summarization tasks to investigate potential reasons for this . authors leverage abundance of data from reading comprehension datasets into longer-premise NLI datasets to improve their models . |
| Outcome: | The proposed models outperform models trained on converted datasets due to the difference in premise lengths. |
More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling (2024.findings-naacl)
Copied to clipboard
Bingsheng Yao, Guiming Chen, Ruishi Zou, Yuxuan Lu, Jiachen Li, Shao Zhang, Yisi Sang, Sijia Liu, James Hendler, Dakuo Wang
| Challenge: | Existing studies on LLM prompting focus on selecting a better set of data samples inside one single prompt input, but why not design and leverage multiple ICL prompts together to further improve the LLM’s performance? |
| Approach: | They propose a low-resource LLM prompting technique to optimize the construction of multiple ICL prompt inputs to produce confident predictions. |
| Outcome: | The proposed technique can produce confident predictions by optimizing the construction of multiple ICL prompt inputs on four NLI datasets and one QA dataset. |
It is not True that Transformers are Inductive Learners: Probing NLI Models with External Negation (2024.eacl-long)
Copied to clipboard
| Challenge: | et al., 2017) show that NLI models learn to treat external negation as a distractor . e-learning models fail to inductively learn the role of negation for NLI tasks . |
| Approach: | They propose that models fine-tuned on NLI datasets learn to treat external negation as a distractor, effectively ignoring its presence in hypothesis sentences. |
| Outcome: | The proposed models learn to treat external negation as a distractor, the authors show . they also fail to inductively learn the law of the excluded middle for a single prefix . |
To What Extent Do Natural Language Understanding Datasets Correlate to Logical Reasoning? A Method for Diagnosing Logical Reasoning. (2022.coling-1)
Copied to clipboard
| Challenge: | Reasoning and knowledge-related skills are considered as fundamental skills for natural language understanding (NLU) tasks. |
| Approach: | They propose a method to diagnose correlations between an NLU dataset and a specific skill. |
| Outcome: | The proposed method is able to diagnose correlations between dataset and logical reasoning skill on 8 MRC and 3 NLI datasets. |
Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to selective prediction are not consistently outperform the simplest baseline MaxProb in all three settings. |
| Approach: | They propose to use a large-scale setup of 17 datasets to study selective prediction in NLP tasks using in-domain, out-of-domain and adversarial settings. |
| Outcome: | The proposed approaches outperform the simplest baseline MaxProb in in-domain, out-of-domain and adversarial settings, but none consistently outperformed in all three settings. |
A Deep Generative Approach to Native Language Identification (2020.coling-main)
Copied to clipboard
| Challenge: | Native language identification (NLI) is a multi-class classification task involving multiple features that capture the systematic fingerprints of the first language in the second language writing. |
| Approach: | They propose a deep generative language modelling approach to NLI that fine-tunes a GPT-2 model separately on texts written by the authors with the same L1 and assigns n-grams to an unseen text. |
| Outcome: | The proposed method outperforms traditional machine learning approaches and currently achieves the best results on the benchmark NLI datasets. |
Unsupervised Natural Language Inference Using PHL Triplet Generation (2022.findings-acl)
Copied to clipboard
| Challenge: | In some cases, training samples may not be available or collecting them could be time-consuming and resource-intensive. |
| Approach: | They propose a procedural approach that leverages sentence transformations to collect PHL triplets for training NLI models. |
| Outcome: | The proposed model outperforms existing models on several NLI benchmarks with a set of sentence transformations. |
Evaluating the Robustness of Discrete Prompts (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods that generate discrete prompts from a small set of training instances have reported superior performance, but manual writing prompts that generalize well is challenging due to several reasons. |
| Approach: | They propose to use discrete prompts to learn lexical constructs that would not be encountered in manually-written prompts. |
| Outcome: | The proposed method is robust against perturbations to NLI inputs but sensitive to other types of perturbations such as shuffling and deletion of prompt tokens. |
Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | Recent advances on abstractive summarization have led to fluent summaries, but factual errors in generated summary still severely limit their use in practice. |
| Approach: | They evaluate summaries produced by state-of-the-art models via crowdsourcing and show that factual errors occur frequently. |
| Outcome: | The proposed models can detect errors and reduce them by reranking alternative summaries. |
Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Natural language inference (NLI) tasks are difficult to perform on large datasets . a small number of simple sentences can improve model performance, authors say . |
| Approach: | They propose to use syntactically simple sentences to test the inference ability of NLI models. |
| Outcome: | The proposed set of simple sentences shows that the models fine-tuned on MNLI and SNLI perform poorly on Simple Pair. |
Repurposing Entailment for Multi-Hop Question Answering Tasks (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to use entailment models for question answering are limited . large scale datasets are typically framed at a sentence level, whereas question answering requires verifying whether multiple sentences, taken together as a premise, entitle a hypothesis. |
| Approach: | They propose a general architecture that can use entailment models for multi-hop QA tasks. |
| Outcome: | The proposed model outperforms QA models trained on target datasets and the OpenAI transformer models. |
Can NLI Models Verify QA Systems’ Predictions? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent question answering systems perform well on benchmark datasets, but are not always well-calibrated to spot spurious answers under distribution shifts. |
| Approach: | They propose to use natural language inference to verify whether answers are correct . they leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules. |
| Outcome: | The proposed approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. |
Partial-input baselines show that NLI models can ignore context, but they don’t. (2022.naacl-main)
Copied to clipboard
| Challenge: | Researchers have shown that many datasets contain statistical biases, or "annotation artifacts" that systems leverage to correctly predict entailment. |
| Approach: | They propose to use edited contexts to examine RoBERTa models' sensitivity to edited context to examine their model's sensitivity. |
| Outcome: | The proposed model can learn to condition on context, despite being trained on artifact-ridden datasets. |
Validity Assessment of Legal Will Statements as Natural Language Inference (2022.findings-emnlp)
Copied to clipboard
| Challenge: | This study introduces a dataset that focuses on the validity of statements in legal wills. |
| Approach: | They propose a dataset that focuses on the validity of statements in legal wills. |
| Outcome: | The proposed model achieves 80% macro F1 and accuracy, but group accuracy is in mid 80s at best, suggesting that the models’ understanding of the task remains superficial. |
Evaluating Numeracy of Language Models as a Natural Language Inference Task (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have enhanced their capabilities to solve mathematical problems, but other aspects of numeracy remain underexplored. |
| Approach: | They propose to frame numeracy as a Natural Language Inference task to assess the models’ ability to understand both numbers and language contexts. |
| Outcome: | The proposed model outperforms smaller models in arithmetic tasks, indicating that mathematical reasoning cannot be generalized to other numeracy skills such as number comparison and normalization. |
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets are often flooded with repetitive and spurious patterns, leading to a lack of linguistic diversity. |
| Approach: | They propose a method that uses cartography to automatically identify and filter examples that demonstrate challenging reasoning patterns and then automatically compose new ones with similar patterns. |
| Outcome: | The proposed approach improves performance on eight out-of-domain test sets compared to training on the 4x larger MultiNLI dataset. |
SciNLI: A Corpus for Natural Language Inference on Scientific Text (2022.acl-long)
Copied to clipboard
| Challenge: | Existing Natural Language Inference (NLI) datasets are not related to scientific text. |
| Approach: | They propose a large dataset for NLI that captures the formality in scientific text and contains 107,412 sentence pairs extracted from scholarly papers on NLP and computational linguistics. |
| Outcome: | The proposed model achieves a Macro F1 score of only 78.18% and an accuracy of 78.23%. |
PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing systems for Natural Language Inference (NLI) only recognize textual entailment relations on sentence-level . however, even a simple sentence often contains multiple propositions, i.e. distinct units of meaning conveyed by the sentence . |
| Approach: | They propose a system to recognize whether one text is textually entailed by another . they use a corpus of over 45K propositions annotated by human raters to study the textual entailment relation of each proposition in a sentence individually. |
| Outcome: | The proposed dataset can be used to understand the compositionality of NLI labels. |
Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural models pick up on annotation artefacts and spurious correlations, resulting in learning sentences that suffer from the same biases. |
| Approach: | They propose to tackle this problem by using adversarial training to reduce the bias in sentence representations by using an ensemble of adversaries. |
| Outcome: | The proposed approach produces more robust models outperforming previous de-biasing efforts when generalised to 12 other NLI datasets. |
Neutralizing Bias in LLM Reasoning using Entailment Graphs (2025.findings-acl)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) is a foundational understanding task in language understanding. |
| Approach: | They propose a framework to construct counterfactual reasoning data and fine-tune LLMs to reduce attestation bias. |
| Outcome: | The proposed framework reduces hallucinations from attestation bias on original and bias-neutralized datasets while keeping hypotheses unchanged. |
SI-NLI: A Slovene Natural Language Inference Dataset and Its Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets for natural language inference (NLI) are limited to English and a few other well-resourced languages. |
| Approach: | They propose to use a dataset for natural language inference to extend the resources for the task. |
| Outcome: | The proposed dataset is constructed from scratch using knowledgeable annotators with carefully crafted guidelines aiming to avoid common problems in existing datasets. |
LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evidence of human label variation in Natural Language Inference (NLI) however, within-label variation is an additional challenge. |
| Approach: | They propose a linguistically-informed taxonomy for categorizing free-text explanations in English that captures different reasoning strategies behind NLI explanations with a particular focus on within-label variation. |
| Outcome: | The proposed taxonomy can be used to classify explanations in English using a linguistically-informed taxonomies. |